Testing and the Cost of Field Failures

Software reliability is the probability that a program will operate correctly for a specified time in a specified environment. What we may really be interested in is the total cost of faults that remain in the released software and will eventually manifest themselves as failures in the field. We present an approach to estimate failure costs in terms of the input/output behavior of the program and use the estimates to better allocate testing effort so as to reduce the expected cost of field failures. Simulation studies using the partition testing model indicate that substantial benefits may be attainable even if failure rates/costs can only be roughly approximated. The expected cost of field failures can also be combined with measures of the cost of carrying out more testing to lead to better software release decisions.